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The Evaluation Off^i^e of the Region 20 Education ^ 
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(Tej^as) area, is delineated in this report. The introduction 
describes the location, size; and budget of thfe agency as well as 'the 
structure and funding of the evaluation office. This first chapter 
notes the region's futtfre needs for evaluation of microcomputer 
software and computer-related educational' programs and .of the cost 
effectiveness of other .educational program options. Chapter 2 covers 
Region 20 's allocation of reSX>urces, which involves using an hourly 
rate for figuring evaluation support. The e.valuationj^of f ice' s 
interaction with its political environment is treated in chapter 3. 
The. fourth chapter discusses the administrative and program 
evaluation straitegies employed by Region 20 to promote the use of r 
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comments that computer literacy i-s increasingly required of: 
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This paper is part of a symposium '^Managing an Evaluation Unit" 
presented at the 1982 American Education Research Association. Presenters 
are Dr* Tom Saterfiel of the Bureau of Educational Research and Evaluation, 
Mississippi State University; Dr. Hugh Peck/ Research and Development Office, 
Louisiana State Department of Education; Dr. Floraline Steven, Evaluation, 
Los Angeles schools; and Dr. Alan Roecks, Evaluation Services, Education 
Service Center-Region 20, San Antonio. , ■ • 

The first part of the paper provides an introduction to^ the author *s 
setting. Questions answered are: ^ 

1. What is Education Service tenter. Region 20? 

2. How is the Region 20 Evaluation Office set up? 

3. How is the Region 20 Evaluation Office funded? 

The symposium discusses five common purposes faced by a manager or an 
evaluation unit. The. purpo^e^s and. related, questions ar^ as. follows: . 
I. Finance 

1. What kinds of evaluation services will districts want and be willing 
to fund in future years? 
II. Allocation of Resources 

1. What is Region 20 *s method of resource allocation? 

2. What are the administrative advantages of allocating resources based 
^ on an hourly rate? 

3. What are the disadvantages of an hourly rate? 

4. How are -unforeseen requests for evaluation support handled? 
III. Politics 

. 1. Wha^t is jthe roleof^the Evaluation-Office In- the poetical environment 
of the. Setvice Center? 
2. What is the rdle of Regioh 20 Evaluation in the politics of districts 
it serves? . ' 
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IV. Use of Evaluative Information (Dissemination) 

1. What administrative strategies does Region 20 employ to promote use 

of evaluations? 

I . ■ 

2. What program evaluation strategies does Region 20, employ to promote 
use of evaluations? 

V* Personnel « 

1. What kind of personnel are needed to carry out evaluation ,w6rk in 
an intermediate education, service agency? 

2. ' What are the special requirements for* the, head of an evaluation 

office in an intermediate agency? 

3. What are the future requirements for evaluation personnel? 



This report vas illustrated by Mariffn BraHy^v Her patience and 
creativity are appreciated • 
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Regvon 20 - 
What is it??? 



Introduction 

What is Education Service Center^ Region^ 2V? 

The State of Texas is divided into 20 regions served by intermediate 
education agencies. Established as part of the Public Education System of 
Texas in 1967, th^ service agencies offe^x school districts and teachers an 
• opportunity to receive specialized services 
which normally would be beyond the reach 
of the average-sized school system. By 
participating in -programs with intermediate 
agencies, school systems take advantage of 
the cost saving benefits and progiram financing 
and, planning which- result from cooperative 
efforts. They also realize additional funding 
and services that are available from state and 
federal services for regional programs, of this type. 

One of these 20 Texas centers. Education Service Center-Region *20, 
or simply Region 20, located in San Antonio, provides services and programs ^ 
to 50 school districts in a 14-county area of South Texas. In 1981, the 
Ceater provided over 50 program and services, many funded from state and 
federal competitive grants. The Center had a $9, 000, OOtf budget and employed 
over 400 personnel. , • 

How is the Region 20 Eoalmtion Office set up? 

In ll977^ Region 20 set up ah Evaluation Office. The initial function^ 
of the office was to evaluate all Center programs and services and to provide 
feedback on the effectiveness of these offerings. The office later adc^ed a 
second function, providing evaluation services to districts on a cost recovery 
basiy. In 81-82, the office was made up of^ 12 full-time professional personnel. 
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several part-time professionals and a strong cadre of clerical staff, many 
s of whom were part-time. The Evaluation Office has^a coordinator and three 
work units. One work unit evaluates programs delivered to districts by 
Center staff, another provides evaluation services' to districts and a third 
provides evaluation services to both the Center and districts. Within each 
work unit ap^^ manager, an evaluator, evaluation assistant (s) , and clerical 
support. Projiact evaluation is carried out as a team effort capitalizing 
on the various roles and abilities of available staff. ^Evaluation staff 
record time spent on each project. ^ ^ * 

V How is the Region 20 Evaluation Office funded? 

Funding for evaluation is from three sources. The largest source is 
federal and state competitive grants for Center programs provided tp districts. 
Evaluation looks at the effectiveness of these teacher training programs. 
Since 1977, Region *20 has carried out 180 evaluations for Center projects" 
totalling $15,000,000. A smaller amount of funding comes from special ^ 
projects. These include technical assistance f or ^chdols having Title I 
migrant programs, a research study> on the effectiveness of use o'f evaluative 
information and direct evaluation support for two district based Center 
programs. ' A third source is contract work with sonie of the larger school 
^districts. How the office's 12 full-time evaluation staf f| are funded is 
depicted below. 



^^^^^^^^ 
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year p.o^ses to be . year of cutbacks fo. u,to. 20. Most 
feda„l competitive grants win be lost due to changes iu uational policy. 
The Evaluatiou OHice can lose up to six evaluatots i, additional' funds " 
«e not identified. «e are optimistic about tA future. Districts, once 
fa^Uat „itb out services, seem interested in receiving evaluation support. ■ 
ErfSrts are „der„ay to ma.e our services .re visible to districts in bopes 
of making up the projected deficit. 
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- <i I. Finance 

Wiat kinds of evaluation services will districts want and be willing to 
\ 

fund in future years? 

Distiricts will need support in how to use microcomputers. First, 
there is a^need to evaluate the effectiveness of educational courseware 
for microcomputers. Some frequently asked questions include, "How 
consistent is the courseware with the existing curriculum? 
Do objectives of the courseware coincide, with th^ 
objectives being taught? Are instructional 
strategies coQSistent: i.e., does the course- 
ware use the same instructional approach' 
employed by the classroom teacher?" 
Second, evaluation offices can assist ''Those Were the good old days." 

^implementing computer managed instructional systems that manage classroom 
data.^ Some systems, such as the nationally validated Comprehensive 
Achievement Monitoring out of Hopkins, Minnesota, help lurprove teacher-made 
tests. This is done by using itei|j'^statistics stored by the microcomputer. 

• Future years will cause education to becoij^ more accountable. Data 
on the cost of educational services will be required. The role of evaluation 
may be one of management consultant. The evalUator may be asked whdch 
program options are- most effective. Common questions include "How can a 
department best be reorganized? Should we conttkct for school bus service 
qr purchase our own buses?" , ^ 
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II. Allocation of Resources 

What is Region^20^s method of resource allocation? 

Allocation of resources is made easier by having an hourly rate for 

r 

evaluation support. This method is required by law as we provide support 
on a contractual basis to 50 separate projects. Projects are filled as 
they receive services. The rate is set at the beginning of each year an4 
•^s calculated by dividing estimated revenue by available staff time. 
(See Roecks & Flores, 1980). Managers of the three evaluation units use 
the hourly rate to determine the amount of time to be put in for each 
project. A, talile giving the amount of time; available for ea^h position is 
used to determine how much staff time can be put in. (See Appendix A), 
Time a^vailable is then matched to ^he amount of work to be put in for each 
management lanit. ^ 
What ar^ the advantages of allbcating resources based on an hourly- rate? 

First, each project receives evaluation support accprding to its level 
of funding. Second, information is available on when th^l^valuation work 
was completed, who did the evaluation and the kind of evaluation activity 
performed. Third, projects can benefit from evaluation work done on other - 
similar projects. Efficiency is accomplished by grouping like evaluation 
tasks^ For example, this year we have three programs in the area of English 
composition. We have 'grouped the projects and assigned them to one evaluate!^ 
Common tasks are identified and additional evaluation time* is made available^ 
for the three projects as a whole. Also, there is a senendipitous outcome -for 
project staff as a result of evaluation grouping: project staff are forced to 
communicate among themselves, with the evaluator serving as facilitator^ A 
fourth benefit "-is that the system allows bettejr use of staff skills. Any 
evaluation staff is ''eligible^^ to work on any project. Hence it' is possible 
to provide a wider array of evaluation support for project^* 



l/hat caie some of the disadvantages of an hourly ram? 



"Which evaluation 
should I work on next: 




A major difficulty is staff have to keep their hours weekly. The amount <. 

> 

of paperwork is then increased. Valuable staff time is lost. Hours are recorded 

and coded into a microcomputer. At l^ast 20 hours per week of clerical support 

is required to maintain the system. Carrying out as many as 50 separate evalua-f 

tions can be an administrative nightmare. As noted earlier, three managerial 

units share this responsibility. Each must organize 

resources to carry out work assigned. Each professional 
« 

staff member has four to twelve 'evaluations . Budgeting 
the right amount of time for evaluation tasks can be 
difficult. Unforeseen circumstances pan mean difficulty 
in carrying out promised evaluations. The system allows* 
very little latitude; evaluations need to be done on time 
Professi6nal staff can charge no more than 40 hrs./week 

of time. Often extra tj-me must be put in outside the regular work week. A 
final problem is explaining the hourly rate,^ especially to district clients. 
We find it, best to list the kinds of evaluation products to be received. 
How are unforeseen requests for evaluation support handled? 

Politically, these requests cannot be denied. They have to be carried \ 

4 

out, usually under tight deadlines. Several strategies have been successfully 

employed. They involve setting aside 
resources for ad hoc seirvices durirlg 
initial planning. One strategy is to 
dedicate a portion of a. person* s time 
to such requests. AnotKelr stratjpgy, 
used for the last two yeajs, sets aside 
up' to 10% in each eyaluation plan for 
ad hoc services, hoc services are 
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carried out by the ^Valuator assigned to project person making the request. 
Carrying out unexpected reque&ts is made easier by having part-time 
professional personnel available. Part-time support usually can carry out 
much of the requested work — allowing the office to be minimally disrupted by 
unanticipated reqlie^s, 

III. The Political Environment^ ' 
Whatjis the role of the Evaluation Office in the political environment of 
ie Center? 

Placement of the Evaluation Office within the organization is important. ^, 
Our office is relatively independent of. the programs it evaluates. (Anderson & 
Ball, 1978). OrganizatioYially , the coordinator is responsible to the 
Executive Director; fiscally, the office manages its own monies. This 
arrangement has allowed the office to be seen as somewhat objective. Main- 
taining this independence has been difficult, (Roecks, et al, 1979). When 
faced with controversial data, some project staff have tried to bring the 
bearers of bad news — the evaluator — under their control. Others have attempted 
to control the level of evaluation funding. Suppjprt from the top was necessary 
to withstand these , challenges. 

Most of any political success our office has had has been facilitated 
by Center's Executive Director, During our first three years (1977-80), we 
experienced many chadlenges to our funding. (See my 1980 Evaluation Network 
paper "Resistance to Evaluation" (Roecks, 1980)). The ExecutlvelDirector 
was able to educate the funding agencies on how evaluation^ 
was used to monitor program activities and to improve the . 
air overall program JthrusCi, , He was also able to gain support 
from various audiences for evaluative information. This 
allowed the ^evaluations to have a hearing in the political J 
arena. Meetings were s^t up where evaluation findings 
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were reviewed by those affected. Included were^regram clients, funding 
agency personnel, board members of the funding and recipient agencies and 

0 

administration from districts served. Credib^^ty was enhanced by having 
program* rather than evaluation personnel explain findings. 

What is the vole of Region '20 Evaluation in the politics of districts it 
serves? 

/ • 

The Service Center must function in an advisory capacity to schools. 

V 

* r 

It cannot be effective if it tells administrators how to i;un tlieir districts. 

The Evaluation Office, as part of the Center, must 
* 

exerc'ise caution in reporting evaluation findings. 

Unless pemission is "given, evaluation findings. are 

not shared with other districts. When negative findings 

result, they are reported in conjunction wit-h 

recommendations. If at all possible, recommendations ^ 

are made joint Ivh^J evaluation and program personnel. 

This practice tends to "lessen the blow"^f the 

sometimes unexpected, negative results. 

What kinds of evaluations do we carry out? The focus of our evaluations 

is on specific federal prograrjfi. We seldom look at the overall focus of the 

district. This limits the impact of evaluations. We have carried out more 

general evaluations for a few districts. Here our primary audiences were 

the superintendent and school board. In one district! where we have the * 

« interest and support of the superintendent a nc^ oard^ the evaluations have 

been used for program improvement. 

IV. Use of Evaluative Information (Dissemination) 

What administrative strategies does Region 20 employ to promote use of - 

evaluations? ^ ^ 

• Hire people with good communication skills. The statistical expert 
« 

who program staff could not understand does not^it well in our offic 




• Budget time^for using evaluation data/ We try to set aside about 10%. 
^ Planning for better use is an ongoing activity. Last year we were 

able to get a planning document funded. The small scale [NIE gr'ant 
looks at the amount of time spent by evaliiators in using evaluation 
data and the overall level of use* ' i 

• Documentation of use of evaluative data is required in each annual 
final report. (See Roecks & Casper, 1980). 

. What program evaluation strategies does Region 20 employ to promote ^use of 
evaluations? « * • 

Involve decision maker^ early. Get them to commit- tp a written 
evaluation plan. 

• Focus. on areas that are priority to the decision maker. 

• Decisioii makers prefer qualitative^ data . Interview information 
is more credible than tables of numbers. 

• Report 'findings informally. Written communication is usually ^ 
less effective. ^ . ' ' 

• I^ results are negative, talk results over with project staff. Report 
. negative* results In conjunction with recommendations developed 

cooperatively by project and evaluation personnel. 

V. Personnel • 
What kind of personnel are needed to carry out evaluation work in an intermediate 
education servtce agency? ' I 

In comparison to other educational (operations, more secretarial and ^ ^ 

» r 

.clerical support- is required in the research and evaluation area: For our 
office, two to four hours of support are allocated for each professional 
position. A part-time person whose primary task is transcribing interview 
information and a ftill-time component secretary are also employed. 



Keeping secretarial help used to be a problem. Turnover for fullr 

. time persoii^l vas high. Staff went to pther departments where the 

work load was' less demanding. The problem has been solved by employing a 

% 

strong cadre of ^ part-time secretaries. Peaks in work loads can be adjusted 
with available personnel. This solution has resulted in higher quality, 
secretarial support. 

At the professional staff level, we need people who .have good qualitative , 
skills, who can wr^te and are good communicators.. Entry level is usually a 
Master's degree. People trained in school psychology or guidance /|ind 
counseling with graduate record examination scores above 1000 have worked out 
well. Background in special education, curriculum design or bilingual education 
is desirable. Several staff have expertise in statistics and computers. 
fv%at ore the requirements for the head of an evaluation office in an 
intermediate ^agency? ' 

^ack^ound and experience in administration are requisites. Being 
familiar with budgets is especially important. Estimating the cost of 

n 

products to be delivered to districts proves to be a continuing challenge. 

The process of negot^.ating contracts can be difficult. » , 

^ The administrator must be prepared to dedicate a larger than normal 
amount of time to reinforce staff for good work. The profession of program 
evaluation' does not provide a great deal of positive reinforcement for its 
members. Evaluators ge'nerally deal with problems and ways to improve the 
present state of affairs. As a result, turnover can be high and evaluators 
become /'burned out". The head of a R&E office must keep this in kind; he 
or she Heeds to provide a reward system within the office environment. 
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W?2at are future requirements for personnel in an intermediate education agency? 

The office is moving towards literacy in microcomputers. Support 
staff are learning how to use word processing. Professional, staff are given 
the opportunity to become involved with microcomputers. Whenever possible, 
evaluation work is identified so that the microcomputer can be used. Program 
staff then gain an understanding about the microcomputer while working with 
their evaluator. * 
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Appendix A 

Time Account^ility for 12 Month Fyll-Time Evaluation Lmpioyees^ 



Evaluation 
Position 



Allocation of Available Time^ 



Leave 



Evaluation Time 



„ , . K o . Avail Avail Hours % f 

Holidays^ Vacation^ Sick^ inSvc^ Days Hours Monthly Chargeable 



Coordinator 


18 


1,2 


9 


,10.0 


Manager 


18 


12 


9 


8.5 


Evaluatpr* (SP) 


18 


12 


. i' 


7.5 


Evaluator 


18 


. 12 


9 


7.5 


Evaluation Asst II 


18 


12 


9 


7.0 


Evaluation Asst I 


18 


. • 12 


9 


6.5 



212.0 
•213.5 
214.5 
214.5 
215.0 
215.5 



1696 
^708 
1716 
1716 
1720 
1724 



141.3- 

142.3 

143.0 

143.0 

143.3 

143.7 



60% 8 
60% 
40% h. 
86% 
85% 
85% . 



84.8 
85.4 
57.2 
114.4 
121.8 
122.1 



^ The number of days in year (365) Jess weekend days (104) equals 261, 
b The number of ESC-20 holidays during the schoo*l ■ year . 
'^ The number of authorized vacation days. 

<l'The average number of sick days based upon 1977-78 attendance. 

« Based upon ESC-20 policy and procedures for training activities. 

^ Percentage is based on previous years' time "and effort records. 

8 50% of. Coordinatolrs ' -time is prorated for administration. The 
remaining 10% represents time that maybe charged to a project 
V7hen carrying out» specific evaluation activity. 

Special Projects Evaluator works on numerous non-chargeablfe 
activities for an estimated total of about 40% chargeable. 



Ste2^ Divide the estimated revenues by the estimated annual chargeable hours 
to obtain the hourly rate. 



Hourly » Revenue Av^ilable 

Hours chargeable ^ contracted hours 



Example 
Hourly Rate 



$^242,500 
13,698.3 



Hourly Rate « $17.70> 
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Hours 





Contracted Hours 


0 


1 


Coordinator 


1017.6 


3 


Managers 


3074.4 


1 


Evaluator (SP) 


686.4 


2.5 


Eval' Asst II (11 mo.) 


2679.6 


1 


Eval Asst I 


1465.2 


1 


Eval Asst 1 (11 mo.) 
J / 


1343.1 



Appendix A ContM. 



TASK 
CODE 

1 H££DS ASSESSWm 

si 

* . Cootulcaclon vlcb Canctr per»oiinftl dtolrlng dact. 
. Flic ftemrch for prtvloutly obctlnfd dtct. 

. J^rtptratlon of dACt coUtcclng InftCnsMnCft. 
.''Stlccting atopla^ ^ll«cclng d^ct. 
. Tabulating, procstaiag 4 analyflng rtaulcy/N^ 
. Report prtptritlon. \^ " / ^ 

* . Conaulcatlona to int«r)prctf > explain findings. 
. Adnialatratlvef4 Supervising. 

CHARCIABLE : If related Co funded project 

H0fi-CHARG£AX1£ : If a new propoaal. 

2 PROPOSAL D£VrJ)P>gNT 

* . Conferencea, aeetinga, 4 dlacuaalons. 
. Reviewing objectlvaa. 

. Preparing' the evaluation aectlon. ' 

. Reading .or Tev4.cving propoaala for conent. . 

. Writing complete propoaal i 

. Adalniattatlve 4 SGperviglng. 



TASK 
CODE 



CHAKGEABL£: 



If a continuation propoaal. 



^ lk)y''CRARCZABLE ; If a neiv propoaa> 



3 • , lEVALUATXOH FUKKIHC , 

. Consultation vlthln rvaluatlon ataff. 
. ?rap4irl&g tbe ^valuation Plaxi. 



CHARCEABU; 



If related to funded project* 



DXTA COLLECTION AND PROCESSIHC 

Developing or aelectlng Inatruoanta. 
Travel tlae connected vlth collecting deta. 
Conducting Intervleva; telephone or In^peraon. 
Obaervatlona; workahopa* acboola, centera, etc. 
Adnlniaterlng exaaa^ teata, quaatlonnalrea, etc. 
Dlatrlbutlng 4 Collecting queaCionn^rea. 
Brleflnga pertaining to data collection. 
D«valoping a record keeping iyatea* " 
Processing workshop ev&luatlon foru. 
Tabulating or scoring other Instroment data, 
•rspsrstlon for csrd punching 4 covputer run. 
Ststlstlcsl snalysls of dsts. 

Reviewing records, reports, or other infonutlon. 
Ad&inlstratlve 4 Supervising. 



REPORTING EVALUATION 



Writing findings, memorsnda, \pcerln, or flnsl reports. 
Dlsseninatlng evaluation reauita. 
Adclniatrative & SupervlalngJ 



CHARGEABLE: 



CHARGEABLE; 



If related to funded prQject. 



If related to funded project. 



6 TECKKICAL ASSISTANCE aEA) 

. Travel tizoc connected with providing aailatanca. 

. Conducting or p&rtlcipatlng in worVcshopa. 

. Other asalacancc provided. 

. Adnitiiscrative 4 Supcrvlalng. 



CHARGEABLE: 



If related to a funded project. 



IKTERACTION^ WITH PROJECT STAFF ^ 

. Meetings ixwolving learning about program. 

. Contact abopt developing evaluation plan. 

. Briefings, conferences, & slgn«*off of plana. 

. Verbally conveying f Indlnga or resulta of evaluation. 

. Providing technical consultation. 

. Formulating recosscendatlona. ' 

. Conaultatlons Ihvolving ualng evaluation data. \ 

. Adoiniatrative 4 Supervlalng. 



CHARGEABLE: 



•If ralated to a funded project. 



8 INTERNAL ;FROCEDgR£$ 

/ 

. JUl inscrvlce session 4 retreet. 
. Canter, component, 4 unit neetings. 
. Ares conferences 4 workshops. 
. Special atudlea 4 'ptojecta. ^ 
. Preparaclon of profaaaional papera. 
. Screening appllcanta for enploycvent.. 
. Attendance at profeatlonal Dsetinga. 
. AdminlatraClve 4 Supervising. 



CHARGEABLE ! 



11 ALL ABSOICES 

. Vacation 
. Sick 
. Peraonal 
«• Jury /military 

• Other 

CHARGEABLE : 



If related to s fundsd project. 
Prorsted^{or personnel vith split 
f \tndints . 



Prorated foi^ peraonnel vith 
apllt funding. 



Exclusive of direct interaction vith Project .^taff . 
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